1. Field of the Invention
The present invention relates to an apparatus and method of three-dimensional modeling, and a medium on which a three-dimensional modeling program is recorded. More particularly, the present invention relates to an apparatus and method of three-dimensional modeling for generating a three-dimensional model of an object of interest, and a medium on which a three-dimensional modeling program is recorded.
2. Description of the Background Art
Recently, various three-dimensional scanners that input the shape and texture (color and pattern) of a three-dimensional object of interest are developed for the purpose of obtaining a three-dimensional graphic image.
For example, Japanese Patent Laying-Open No. 5-135155 discloses a three-dimensional modeling apparatus that generates a three-dimensional model of an object of interest by shooting the object of interest mounted on a turntable with a monocular camera to obtain a plurality of silhouette images. According to this apparatus, a plurality of cross sectional shapes of the object of interest in a plurality of horizontal planes (plane perpendicular to the axis of rotation of the turntable) on the basis of the obtained plurality of silhouette images. Then, the points on the contour line of the shape of the object of interest in adjacent horizontal planes are joined together to form a triangular patch, from which a three-dimensional model of the object is produced. In such a three-dimensional modeling apparatus, the camera position can be fixed since the object of interest is rotated on a turntable. It is therefore not necessary to calculate the camera position.
However, such a turntable is extremely costly, which has become the bottleneck in realizing a simple and economic three-dimensional modeling apparatus.
A method of calculating the camera position from an object image obtained by shooting an object of interest using the commonly called factorization method is disclosed by Takeo Kanade, Conrad J. Poelman, and Toshihiko Morita in xe2x80x9cA Factorization Method for Shape and Motion Recoveryxe2x80x9d Transaction of Institute of Electronics, Information and Communication Engineers of Japan, D-II, Vol. J76-D-II, No. 8, pp. 1497-1505, August 1993. However, this method is inadequate to generate a full model of an object of interest since there was a problem that a calculation error is sequentially accumulated.
In view of the foregoing, an object of the present invention is to provide a simple and economic apparatus and method of three-dimensional modeling that does not require a turntable, and a medium on which a three-dimensional modeling program is recorded.
According to an aspect of the present invention, a three-dimensional modeling apparatus generates a three-dimensional model of an object of interest, and includes a plurality of reference subjects (objects), an image capture device, a calculation device, and a generator. The plurality of reference subjects are arranged at predetermined positions. The image capture device, a camera for example, picks up the image of the object of interest together with the reference subject to obtain an object image. The calculation device refers to the reference subject in the object image to calculate the shooting position of the image capture device. The generator generates a three-dimensional model according to the object image and the shooting position.
Preferably, the reference subjects are arranged irregularly and asymmetrically. Preferably, the reference subjects are arranged in a scattered manner. More specifically, when the distance from one certain reference subject to another reference subject is l, and the angle between two direction vectors from that certain reference subject to two other reference subjects is xcex8, it is desired that the arrangement of the reference subjects is determined so that the distribution of the number of reference subjects at distance l and the distribution of the number of reference subjects with angle xcex8 are uniform. Reference subjects arranged regularly can be provided in addition to reference subjects arranged irregularly.
Preferably, the plurality of reference subjects have a plurality of attributes. When the reference subject has an attribute, it is desirable to determine the arrangement of the reference subjects so that the distribution of the number of reference subjects at distance l and the distribution of the number of reference subjects with angle xcex8 are uniform for each attribute.
Preferably, the attribute is a color. When the plurality of reference subjects have a plurality of color attributes and the hue of 360 degrees at a predetermined color space is divided into that plurality of colors, the plurality of colors are preferably selected so that the angle between two adjacent colors in the divided hue is as large as possible.
Preferably, the reference subjects are formed on a 2-D plane e.g. a reference sheet. However, the reference subject can be formed on a plane of a plate thicker than the reference sheet, a platform, or the floor.
The reference subjects are formed at a reference object having a stereoscopic structure. Here, an object having a stereoscopic structure covered with the reference sheet can be used as a reference object.
Preferably, the calculation device calculates the shooting position according to the Hough transform method.
Preferably, the shooting position calculation device includes an extractor, a selector, an allocation device, a candidate calculation device, an evaluation device, and a determination device. The extractor extracts a shot reference subject from the object image. The selector selects a plurality of reference subjects from the extracted reference subject. The allocation device establishes correspondence between each of the selected reference subjects and any one of the arranged plurality of reference subjects (the position data of which are stored to the allocation device) to allocate the position of a corresponding reference subject as the position candidate. The candidate calculation device calculates the shooting position candidate of the capture device according to the position candidate of the reference subject. The evaluation device applies an evaluation value (evaluation based on probability, for example points) to the shooting position candidate. The determination device determines the shooting position candidate having the greatest evaluation value out of the calculated plurality of shooting position candidates as the shooting position for the image capture device.
Preferably, the plurality of reference subjects have a plurality of attributes. The shooting position calculation device includes an extractor, a selector, an allocation device, a candidate calculation device, an evaluation device, and a determination device. The extractor extracts a shot reference subject from the object image. The selector selects a plurality of reference subjects from the extracted reference subject. The allocation device establishes correspondence between each of the selected reference subjects and any one of the arranged plurality of reference subjects (the position data of which are stored to the allocation device) having the same attribute to allocate the position of a corresponding reference subject as the position candidate. The candidate calculation device calculates the shooting position candidate of the image capture device according to the position candidates of the reference subject. The evaluation device applies an evaluation value (evaluation based on probability, for example points) to the shooting position candidate. The determination device determines the shooting position candidate having the greatest evaluation value out of the calculated plurality of shooting position candidates as the shooting position for the image capture device.
Preferably, the reference subject is formed at a reference object having a stereoscopic structure. The shooting position calculation device includes an extractor, a selector, an allocation device, a candidate calculation device, a judge device, an evaluation device, and a determination device. The extractor extracts a shot reference subject from the object image. The selector selects a plurality of reference subjects from the extracted reference subjects. The allocation device establishes correspondence between each of the selected reference subjects and any one of the arranged plurality of reference subjects (the position data of which are stored to the allocation device) to allocate the position of the corresponding reference subject as the position candidate. The candidate calculation device calculates the shooting position candidate of the image capture device according to the position candidate of the reference subject. The judge device judges whether the image capture device can shoot the allocated reference subject from the shooting position candidate. The evaluation device applies an evaluation value (evaluation based on probability, for example points) to the shooting position candidate when judgement is made that the allocated reference subject can be shot. The determination device determines the shooting position candidate having the greatest evaluation value out of the calculated plurality of shooting position candidates as the shooting position for the image capture device.
Preferably, the judge device includes an inner product calculation device, and a judge device. The inner product calculation device calculates the inner product of the view vector indicating the shooting direction of the image capture device and the normal vector of the plane of the reference object where the allocated reference subject is formed. The judge device judges that the image capture device can shoot the allocated reference subject when the inner product is negative; this means the reference subject is visible from the shooting point.
Preferably, the judge device includes an occluded region calculation device, and a judge device. The occluded region calculation device calculates a conical occluded region (the region assumed to be blocked by the reference subject from the shooting position candidate) with each predetermined position of the arranged plurality of reference subjects as a vertex and that extends in a direction farther away from the shooting position candidate. The judge device provides judgement that the image capture device can shoot the allocated reference subject when the allocated reference subject is outside the occluded region; this means the reference subject is visible from the shooting point.
Preferably, the evaluation device includes a projection point calculation device, a counter, and a weighting device. The projection point calculation device calculates respective positions of the plurality of projection points on the object image from the arranged plurality of reference subjects according to the shooting position candidate. The counter counts the number of reference subjects located within a predetermined distance from the projection point out of the extracted reference subjects. The weight device applies an evaluation value according to the number of counted reference subjects to the shooting position candidate.
Preferably, the shooting position calculation device further includes a projection point calculation device, a distance calculation device, and a modify device. The projection point calculation device calculates respective positions of the plurality of projection points on the object image from the arranged plurality of reference subjects according to the determined shooting position. The distance calculation device calculates the distance between the projection point and a reference subject that is in the proximity of the extracted reference subject. The modify device modifies the determined shooting position to reduce the evaluation value according to the distance so that the entire distance becomes smaller, for example, so that the sum of the square of the distance, or the sum of the distance is minimized.
Preferably, the shooting position calculation device further includes a judge device. The judge device provides judgment of a reference subject that the image capture device can shoot from the determined shooting position out of the arranged plurality of reference subjects to use the reference subject judged to be shootable in the calculation by the projection point calculation device, and excluding the reference subject judged to be not shootable from the calculation by the projection point calculation device.
Preferably, the three-dimensional modeling apparatus further includes a setting device for newly setting a reference subject at a position other than the position of a reference subject already in the object image.
Preferably, the setting device erases data associated with a reference subject in the object image and registers data associated with another reference subject.
Preferably, the setting device registers data associated with another reference subject in addition to data associated with a reference subject in the object image.
Preferably, the three-dimensional modeling apparatus further includes a memory and a notification device. The shooting position is stored in the memory. The notification device notifies the person (operator) using the image capture device of a position other than a shooting position already stored in the memory. As a position (the next point of view for shooting) other than a shooting position already stored in the memory (shot point of view), a position remote from the stored shooting position, further preferably a position most remote, is selected. Also, a point of view that can acquire a more accurate shape according to the schematic shape information of the object of interest can be selected as the new shooting point of view.
According to the present invention, the camera shooting position is calculated on the basis of the position of a reference subject in an object image obtained by shooting an object of interest together with a plurality of reference subjects arranged at predetermined positions. Therefore, a three-dimensional model can be generated without using a turntable such as in the conventional case. Thus, a simple and economic three-dimensional modeling apparatus can be realized.
Since the reference subjects are arranged irregularly and asymmetrically, it is not necessary to place the object of interest taking into consideration the critical relationship with the position of the reference subject. Therefore, a three-dimensional model can be generated even when the object of interest is larger than the reference sheet or reference object. Furthermore, even when the object of interest is significantly larger than the reference sheet or reference object, a three-dimensional model thereof can be generated by placing a plurality of reference sheets or reference objects around that huge object of interest.
Since the reference subject has an attribute, the space of search in calculating the shooting position becomes smaller. This means that the calculation speed of the shooting position is improved. Also, difference in the attribute can easily be discriminated since the attribute is a color.
Since the camera shooting position is calculated on the basis of a known reference subject, a calculation error will not be accumulated. Also, since the camera shooting position is calculated according to the Hough transform method, extraction error of a reference subject and effect of noise in the object image can be suppressed. Furthermore, the calculation precision can be modified appropriately.
The calculation accuracy of the shooting position is improved since judgement is made whether a reference subject can be shot or not and a predetermined point is applied to the shooting position candidate only when shooting is allowed.
Furthermore, since the number of reference subjects in the proximity of the projection point from the reference subject onto the object image is counted and a point according to that count is applied to the shooting position candidate, the calculation precision of the shooting position is improved.
Furthermore, since the distance between a projection point from a reference subject onto the object image and a reference subject in the proximity thereto is calculated, and the determined shooting position is modified so that the sum of the square of the distance is minimized, the calculation precision of the shooting position is improved. Furthermore, since determination is made whether the reference subject can be shot or not, and the reference subject that can be shot is used for the projection point calculation and the reference subject that cannot be shot is excluded from the projection point calculation, the calculation precision of the shooting position is further improved.
After shooting a reference object together with the reference subject, a new reference subject is set at a position other than the position of a reference subject already in the obtained object image. Therefore, a three-dimensional model of a huge object of interest can be generated.
The calculated shooting position is stored, and a position other than that stored shooting position is notified to the person picking up the image as the next shooting position. Therefore, an object image required to generate a three-dimensional model can be obtained efficiently.
The foregoing and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.